Reliable, efficient and low cost method for games audio rendering

ABSTRACT

An efficient method for a reliable and accurate rendering of audio in a virtual reality environment, such as computer gaming. It excels other known rendering methods in its ability to achieve high quality audio rendering in real time conditions using a regular personal computer without any need for a special hardware accelerated audio card. A total of k physical properties are defined for the sound source and for the various objects that may affect the generation and distribution of the sound in the simulated environment. For each one of those physical properties some applicable values are predefined in a granularity which is a parameter of the method. This way a k dimensional grid of data points is defined. For each one of those data point the sounds that will be heard by the listener are offline calculated offline as reference samples. When the sound shall be generated in real time, for any given real data point, it is calculated as an interpolation of some near reference samples on the predefined k dimensional grid.

FIELD OF THE INVENTION

The present invention relates generally to a method for rendering audioand acoustic effects in computer games.

BACKGROUND OF THE INVENTION

The level of sophistication and complexity of computer games keepsrising. In particular, the interactive 3D audio visual capabilities ofgames are constantly developing to provide a greater sense of realism tothe gaming experience and to enhance the overall enjoyment of theplayers in single player and multiplayer modes. One of the significantaccelerators of this process has been the stunning development of highlyadvanced surround audio technology for home entertainment and cinemasystems.

The modern audio systems have lead to a demand for rendering of complexacoustic effects in games at a level of realism that is customary in theadvanced movie industry. In order to achieve this, sound synthesisshould depend on many parameters. For example, the sound of a sword thatis swiftly cutting the air would depend on its motion speed as well ason its exact dimensions and shape. As another example, the sound of aball falling on the ground depends upon the ball physical properties,the ground material and the ball speed and angle of hit.

Beyond the task of accurate sound generation, the more challenging taskis simulating authentic distribution of sounds in the game virtualenvironment. This must take in consideration the structure of theobjects that comprise this environment in terms of polygons, theacoustic properties of those objects such as reflection coefficients,the distances among the sound source, the involved objects and thelistener as well as the variation in time of those distances. The abovefactors determine the characteristics of various acoustic effects thatshall be associated with the distributed sound like echo, reverb,Doppler, flange, occlusion, obstruction etc.

The theory of calculating the above sounds and effects that may berelevant to the various scenarios occurring in games are known in theart of acoustics. In particular, there are standard software APIs andlibraries that are commonly used for the complex calculations requiredfor reliable audio rendering. The acoustic calculations can be doneeither offline, during the game creation phase, or in real time, when itis actually played.

The advantage of the offline mode is that it enables accuratecalculations, taking in consideration the various rendering aspects,like the effects of physical properties of objects, the distances amongthem, their dimensions and relevant acoustic affects. However, thenumber of pre-generated sound samples and effects that can bepre-calculated in advance is limited owing to storage capacitylimitations. Therefore, those samples would normally differsignificantly from those required for the dynamically varying real timegaming conditions.

In the real time mode the sound samples and effects are generated in theplayer's computer or gaming console while the game is being played. Theadvantage is that the acoustic parameters may be accurately chosenaccording to the specific conditions at any given moment during thegame. However, as this approach consumes significant processing power,when implemented in a regular PC (Personal Computer), this would come atthe expense of the accuracy of the required calculations, hence limitingthe achieved sense of realism of the gaming experience. This limitationcan be overcome by using dedicated powerful hardware accelerators thatare integrated in advanced audio cards or gaming consoles. However, thissolution implies excess expenses for the player; furthermore thesolution is rigid and cannot be readily adapted for all computer games.

SUMMARY OF THE INVENTION

The background art does not teach or suggest how to achieve reliable andaccurate rendering of sounds and acoustic effects that are required inadvanced games when those are played on regular computers and/or gamingconsoles.

The present invention overcomes these drawbacks of the background art byproviding a method for reliable and accurate rendering of sounds andacoustic effects without requiring excessive processing power. Withoutlimitation, as an example of the many advantages of the presentinvention, the player may enjoy authentic acoustic experience in his orher home entertainment environment while saving the extra expense ofspecial audio cards or gaming consoles, which normally use dedicatedhardware accelerators in order to achieve this goal.

According to some embodiments of the present invention most of theprocessing effort that is required for reliable audio rendering isexpended offline, i.e. in the game creation phase rather than in realtime. This effort is mainly invested in calculating reliable soundsamples for predefined discrete conditions of the game flow. The soundsamples that pertain to the real game conditions are calculated at runtime as interpolation of the off line calculated sound samples.

The offline rendered sound samples are calculated by taking into accountthe relevant parameters that characterize the sound sources, thephysical properties of the objects that take part in the generation ofthe sounds and of the objects that affect the distribution of the soundsto the listening player, and the various acoustic effects that affectthe authentic playback of sound samples to the listener.

The rendered sound samples are preferably calculated at discrete pointsin the multidimensional space that is created from the parameters andproperties for each expected scenario in the game. These rendered soundsamples are incorporated as database records by the game designer in thesoftware package of the game that is supplied to the player.

When the player runs the game in real time on a computer or gamingconsole, for each real combination of values of the parameters andproperties, the accurate rendered sound samples are calculated as aninterpolation of the records that pertain to some points that are closeto the real combination in the multidimensional space. The optionallyallowed discretization error that may result due to the interpolationwould affect the specific interpolation method that is chosen, as wellas the number of near data points that shall be selected. It should benoted that the terms “computer” and “gaming console” are usedinterchangeably and without limitation throughout the presentapplication.

In another embodiment of the invention, if the player computer haslimited processing power, it may optionally send to a dedicated GameServer requests for actual interpolation calculations. The game designertoo may have the option to request that dedicated Game Server foraccurate interpolation calculations in order to save design resourcesand to allow a simpler interpolation method at the player computer.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. The materials, methods, andexamples provided herein are illustrative only and not intended to belimiting.

Implementation of the method and system of the present inventioninvolves performing or completing certain selected tasks or stagesmanually, automatically, or a combination thereof. Moreover, accordingto actual instrumentation and equipment of preferred embodiments of themethod and system of the present invention, several selected stagescould be implemented by hardware or by software on any operating systemof any firmware or a combination thereof. For example, as hardware,selected stages of the invention could be implemented as a chip or acircuit. As software, selected stages of the invention could beimplemented as a plurality of software instructions being executed by acomputer using any suitable operating system. In any case, selectedstages of the method and system of the invention could be described asbeing performed by a data processor, such as a computing platform forexecuting a plurality of instructions.

Although the present invention is described with regard to a “computer”on a “computer network”, it should be noted that optionally any devicefeaturing a data processor and/or the ability to execute one or moreinstructions may be described as a computer, including but not limitedto a PC (personal computer), a server, a minicomputer, a cellulartelephone, a smart phone, a PDA (personal data assistant), a pager, TVdecoder, game console, digital music player, ATM (machine for dispensingcash), POS credit card terminal (point of sale), electronic cashregister. Any two or more of such devices in communication with eachother, and/or any computer in communication with any other computer mayoptionally comprise a “computer network”.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only, and are presentedin order to provide what is believed to be the most useful and readilyunderstood description of the principles and conceptual aspects of theinvention. In this regard, no attempt is made to show structural detailsof the invention in more detail than is necessary for a fundamentalunderstanding of the invention, the description taken with the drawingsmaking apparent to those skilled in the art how the several forms of theinvention may be embodied in practice.

In the drawings:

FIG. 1 illustrates a characteristic game environment, where some gameelements that would affect reliable rendering of the game sounds areparticularly depicted.

FIG. 2 depicts the invented method operation flow.

FIG. 3 depicts a linear interpolation method for the specific case of athree dimensional grid of data points.

FIG. 4 illustrates a general system diagram, comprising a designerserver, a game server and client players

DETAILED DESCRIPTION OF THE INVENTION

A specific embodiment of the present invention is hereby describedreferring to a characteristic game environment that is illustrated inFIG. 1. Some elements that would affect reliable rendering of gamesounds are particularly depicted in this Figure. A falling ball hits theground and a sound sample is consequently generated. This sound sampleis determined by a set of parameters {S1} that characterize hit relatedprimitive sound sample e.g. the ball velocity at the moment of hit andthe hit angle.

The physical properties of the objects that are involved in the hit,such as material type and elasticity, affect the generated sound sampleas well. The relevant physical properties of the ball are denoted by{P1}, and those of the floor are denoted by {P2}.

The generated sound is distributed in the hall space toward the listeneror player as depicted in tick broken arrow. The listener hearsreflections from the walls as well, a phenomenon called “reverb”. Theset of wall properties that affect the reverb, like dispersion andreflection coefficients, are denoted by {P3}. The set of distancesbetween the hit point, the walls and the listener, as well as thedimensions of the wall polygons are denoted by {P4}.

FIG. 2 exemplifies a specific embodiment of the invention for softwarebased method of efficient rendering of reliable computer game sounds,for example by a regular computer. The acoustic environment notations inthe drawing refer to FIG. 1. The upper portion of the drawing describesthe process that takes place at the game development stage while itslower part refers to the game run at the player's home environment.

A typical game is composed of scenarios [201]. Such a scenario isillustrated in FIG. 1. For each one of the scenarios various objects aredefined [202] such as the ball, the floor and the walls in FIG. 1. Foreach one of the objects various physical parameters are specified andare noted in [203] by {P1} . . . {P4}. For each one of the properties, avalue or a range of values are ascribed. Such a range can refer, forexample, to the distances noted in FIG. 1, which may vary depending onthe ball fall location as well as the listener location.

Primitive sound samples [204] are also defined for a typical scenario.In the described scenario this primitive sound sample is the result ofthe ball hitting the ground, which may depend on various parameters{S1}. Those parameters are evaluated depending on possible variations ofthe scenario in real time game conditions. The distribution of the soundin the acoustic environment of a typical scenario may be affected byvarious acoustic affects such as echo, reverb, Doppler, flange,occlusion, obstruction etc {205}. An example of reverb is illustrated inFIG. 1.

Block {206} of FIG. 2 depicts the calculation of a plurality of soundsamples that will eventually be heard by the player, who is the listenerin typical game scenarios. The sound samples may be reproduced via anyspeakers array, in particularly via home entertainment/cinema surroundsystem. Therefore a reliable calculation of the sound sample thatpertains to each one of the surround speakers is required for providingthe player with an authentic spatial experience of the game. Thecalculations are based on common knowledge in the science of acoustics,using common software libraries and APIs. The resulting sound samplesare examined by the game designer who may iteratively adjust thescenario parameters for optimal sound rendering.

The final sound samples are stored in a database [207], typically in WAVformat, for later use in real time conditions. Each sound event isstored as a reference group {S2*} of up to 6 sound samples to conform toconventional home multi speaker surround systems. The calculations areperformed for discrete values of each one of the physical properties, orparameters, that affect the sound rendering. Assuming k such parameters,they form a k dimensional space and a corresponding k dimensional gridof discrete parameter values that we denoted “data points”. Theresulting reference samples are mapped to this k dimensional grid. Theactual granularity of the values that are selected for calculation oneach one of the grid axes is a parameter of the system and is affectedby storage limitations and by the desired accuracy of the soundrendering.

The reference sound samples {S2*} are supplied together with the game PCsoftware package to the game user [209]. While the game is being played,the parameters attain real time values [210] in the k dimensional space,which are typically different from the k dimensional grid values. Inorder to get reliable sound samples {S2}, for any given real time datapoint, as depicted in [212], {S2} is calculated in real time, asdepicted in [211], as an interpolation of some adjacent S2* samples onthe grid. {S2} is then transferred from the PC to any speaker array, inparticular home entertainment/cinema surround speaker system.

The interpolation method that is used in [211] can be any one of themethods that are conventional in the art of multidimensionalinterpolation calculation. Preferably, the method is selected withregard to one or more of grid granulation, the allowed discretizationerror, the desired response time and the expected PC processing power.Some well known interpolation methods that can be used are providedherein as non-limiting illustrative examples, though other methods canbe used as well.

Nearest neighbors: Choosing the n (parameter, 1 by default) nearest (byEuclidian distance) neighbors and average them with the followingformula:

${\overset{\_}{x} = \frac{\sum\limits_{i = 1}^{n}{w_{i}x_{i}}}{\sum\limits_{i = 1}^{n}w_{i}}},$

where each Wi can be one of the following: Wi=(distance₁+1)^((−alpha)),where alpha is also a parameter, 1 by default. Another option is that Wiis e^((−alpha*distancei)).

Linear interpolation: The principle of this method is illustrated inFIG. 3. for the specific Trilinear case, i.e. an interpolation in threedimensional grid of data points. The data points are mapped to thevertices of a normalized unit cube. Consider a unit cube with thelower-left-base vertex at the origin of the grid coordinates as shown inFIG. 3.

The values at each vertex will be denoted V000, V100, V010, . . . V111respectively to the points (x0,y0,z0) . . . (x1,y1,z1). The value atposition (x,y,z) within the cube will be denoted Vxyz and is given by

Vxyz=V000(1−x)(1−y)(1−z)+V100x(1−y)(1−z)+V010(1−x)y(1−z)+V001(1−x)(1−y)z+V101 x(1−y)z+V011(1−x)yz+V110 x y(1−z)+V111 x y z

In general the box will not be of unit size nor will it be aligned atthe origin. Simple translation and scaling (possibly of each axisindependently) can be used.

Iterative interpolation: In each step choose one of the physicalproperties and make a polynomial interpolation only on this axis usingas many points as possible with a Newton form. Given a set of k+1 datapoints (x₀,y₀), . . . , (x_(k),y_(k)) where no two x_(j) are the same,the interpolation polynomial in the Newton form is a linear combinationof Newton basis polynomials

${N(x)}:={\sum\limits_{j = 0}^{k}{a_{j}{n_{j}(x)}}}$

with the Newton basis polynomials defined as

${n_{j}(x)}:={\prod\limits_{i = 0}^{j - 1}\left( {x - x_{i}} \right)}$

while the coefficients are defined by a_(j):=[y₀, . . . ,y_(j)] where[y₀, . . . ,y_(j)] is the notation for divided differences. Thus theNewton polynomial can be written as N(x):=[y₀]+[y₀, y₁](x−x₀)+ . . .+[y₀, . . . , y_(k)](x−x₀)(x−x₁) . . . (x−x_(k−1)).

FIG. 4 exemplifies some variations of the embodiment that is illustratedin FIG. 2. The game creation stages are represented here by “Game DesignServer” [401]. However, the game software and grids of reference soundsamples at relative low granulation data points are transferred to adedicated “Game Server” [402]. Game Server can receive requests foraccurate interpolation calculations from the game designer in order tosave him the burden of accurate calculations and to allow, as well,simpler interpolation calculations at the client player's computer.

Client computer [403] depicts a computer available to a player that is aregular computer, as in FIG. 2. Thin client computer [404] depicts athin client, which may for example be a mobile phone or PDA, or othercomputational device with limited processing power, that is available tothe player. Thin client computer [404] downloads the game software fromthe Game Server 402, not including the grids. Whenever actual audiosamples are required at real time the thin client computer [404] sendsGame Server 402 the real game conditions, Game Server 402 calculates theactual samples by interpolating adjacent reference samples and providesthe thin client computer [404] with the results. FIG. 4 also shows thatoptionally a regular client computer [403] may request interpolationcalculations from Game Server [402] if client computer [403] has limitedprocessing power.

While the invention has been described with respect to a limited numberof embodiments, it will be appreciated that many variations,modifications and other applications of the invention may be made.

1. A method for sound rendering to be used in a computer based virtualreality system comprising: generating a sound sample emanating from asound source; associating one or more acoustic effects to said soundsample for affecting the rendering of said sound sample; predeterminingone or more values for one or more physical parameters for affectingrendering of said sound sample according to said one or more acousticeffects, wherein said physical parameters relate to an interaction of aplurality of objects in the virtual reality system; and interpolatingsaid predetermined one or more values of said one or more physicalparameters to determine an effect on said sound sample.
 2. The method ofclaim 1, wherein said predetermining said one or more values comprisesassociating a total of k physical properties to said sound sample and toany object that may affect perception of said sound sample afterrendering, where k may be any positive number, and defining a kdimensional space thereof; selecting one or more pre-defined values foreach one of said k physical properties to generate a k dimensional gridof data points that is mapped on said k dimensional space; calculatingthe sound sample or samples for any data point in said k dimensionalgrid; and selecting an interpolation method to be applied to said kdimensional grid for calculating the sound sample or samples that wouldbe heard by said listener for any desired combination of k real valuesof said k properties.
 3. The method of claim 2 where the acousticeffects that may affect the rendering of said sound sample in saidacoustic environment may be echo, reverb, Doppler, flange, occlusion,obstruction etc.
 4. The method of claim 2 wherein said physicalparameters comprise one or more of object elasticity, object reflectioncoefficient, polygon dimensions, or relative distances among said soundsource, said objects and said listener etc.
 5. The method of claim 2wherein a plurality of sound samples are calculated to simulate aplurality of corresponding sounds for playback by a multi track audiosurround system.
 6. The method of claim 2 wherein said selecting saidinterpolation method to be applied is at least partially determinedaccording to an allowed discretization error from said interpolation. 7.The method of claim 6, wherein said selecting said interpolation methodto be applied is further at least partially determined according to thenumber of data points for each one of the k physical properties in saidk dimensional grid.
 8. The method of claim 2 wherein at least a portionof the sound samples that are associated with said grid data points arecalculated by interpolating other sound samples on said grid that wereaccurately calculated beforehand.
 9. The method of claim 2 wherein thevirtual reality system comprises a computer game, wherein the pluralityof objects are provided through said computer game and wherein thelistener comprises at least one game player.
 10. The method of claim 8wherein said calculating the sound sample or samples for any data pointin said k dimensional grid is performed before game play such that aplurality of precalculated multidimensional grids are provided with saidcomputer game, and wherein said calculating said interpolation at thegame real time conditions is done by the player computer for any actualcombination of values of physical properties that pertain to any realsituation during the game.
 11. The method of claim 9 further comprising:following any event of new sound samples generation, forecasting themost probable conditions for the next event of sound generation, andreading one or more new predefined sound samples that are not yet cachedin the dynamic memory of the computer.
 12. The method of claim 8 whereinsaid calculating the sound sample or samples for any data point in saidk dimensional grid is performed before game play such that a pluralityof precalculated multidimensional grids are provided with said computergame, and wherein said calculating said interpolation at the game realtime conditions, for any actual combination of values of physicalproperties that pertain to any real situation during the game, is doneby a dedicated game server to which the player computer connects andrequests the required sound samples when actually required.
 13. Themethod of claim 11 wherein the player computer is a thin client such asa mobile phone or a PDA (personal digital assistant).